fix: resolve quiesceAndThenRestartMixedOps() flakiness by AlexKehayov · Pull Request #24567 · hiero-ledger/hiero-consensus-node

AlexKehayov · 2026-03-26T21:58:32Z

Description:
Follow-up to #24404.

Root Cause:
The original test's log assertion (assertHgcaaLogContains) searched the entire log file and trivially matched a transient startup quiescence event from node boot. It never validated that real sustained quiescence occurred during the test. The rare CI flake was the balance assertion failing due to tight timing margins.

Findings:

Transient vs real quiescence: The network produces ~5-6 quiescence transitions per run. All but one are transient (<1ms between "Started" and "Stopping"). The real one lasts ~22s - the heartbeat runs until it finds the TCT near the scheduled tx expiry.
Staking period transactions prevent quiescence: With staking.periodMins=1 (test default), staking transactions fire every 60s of consensus time, keeping the network active. Suppressing them is required for reliable quiescence.
overridingAllOf doesn't survive restart: It updates system file 0.0.121 in state, but env overrides set via restartAtNextConfigVersion(envOverrides) are set as process environment variables that take effect at startup - before the system file is processed.

Changes:

LogContainmentPairTimeframeOp.java (new) - Generic UtilOp that finds two log patterns in order within a timeframe, with a time gap in [minGap, maxGap]. Distinguishes real state transitions from transient flickers.
UtilVerbs.java - Added assertHgcaaLogContainsPairTimeframe() verb.
LifecycleTest.java - Added restartAtNextConfigVersion(Map<String, String> envOverrides) overload that passes env overrides through to the subprocess nodes.
QuiesceThenMixedOpsRestartTest.java - Uses restartAtNextConfigVersion with env overrides to suppress staking transactions. Replaced single-pattern log assertion with paired assertion (minGap=5s, maxGap=40s) that deterministically matches only the real sustained quiescence.

Fixes #24398

trunk-io · 2026-03-26T21:58:35Z

Merging to main in this repository is managed by Trunk.

To merge this pull request, check the box to the left or comment /trunk merge below.

lfdt-bot · 2026-03-26T21:58:49Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

gkozyryatskyy

LGTM! Thank you @AlexKehayov !

...in/java/com/hedera/services/bdd/suites/regression/system/QuiesceThenMixedOpsRestartTest.java

codecov · 2026-03-26T22:11:12Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@             Coverage Diff              @@
##               main   #24567      +/-   ##
============================================
+ Coverage     78.23%   78.29%   +0.06%     
+ Complexity    11692    11688       -4     
============================================
  Files          2490     2486       -4     
  Lines         95107    94900     -207     
  Branches      10293    10280      -13     
============================================
- Hits          74406    74305     -101     
+ Misses        16960    16855     -105     
+ Partials       3741     3740       -1

see 19 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2026-03-26T23:02:11Z

Node: HAPI Test (Misc) Results

160 files ±0 3 errors 157 suites ±0 1h 16m 46s ⏱️ - 16m 18s
170 tests ±0 168 ✅ ±0 2 💤 ±0 0 ❌ ±0
1 222 runs ±0 1 220 ✅ ±0 2 💤 ±0 0 ❌ ±0

For more details on these parsing errors, see this check.

Results for commit 797302a. ± Comparison against base commit bc67352.

♻️ This comment has been updated with latest results.

…to fix quiescence Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

… 24398-quiescence-follow-up

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

codacy-production · 2026-03-31T12:16:47Z

Not up to standards ⛔

🟢 Coverage ∅ diff coverage · 0.00% coverage variation

Metric Results

Coverage variation ✅ 0.00% coverage variation (-1.00%)

Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details

Coverable lines Covered lines Coverage

Common ancestor commit (3d9574f) 94803 77996 82.27%

Head commit (583cd04) 94803 (+0) 77994 (-2) 82.27% (0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

Coverable lines Covered lines Diff coverage

Pull request (#24567) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

_{TIP This summary will be updated as you push new changes. Give us feedback}

# Conflicts: # hedera-node/test-clients/build.gradle.kts

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

… quiescence logic

github-actions · 2026-04-02T16:36:18Z

⚠️ New Flaky Test(s) Detected — Action Required

New flaky tests were detected in this run. New tickets have been created and require your attention — please investigate whether the flakiness was introduced by changes in this PR.

Test	Ticket
`com.hedera.services.bdd.suites.integration.BlockNodeRewardsTests#activeNodeWithOneRegisteredBlockNodeGetsConsensusAndBlockReward`	#24711 (🆕 New)

Please review the linked tickets and determine if any of the new issues were caused by your changes.

AlexKehayov self-assigned this Mar 26, 2026

AlexKehayov requested review from a team as code owners March 26, 2026 21:58

gkozyryatskyy previously approved these changes Mar 26, 2026

View reviewed changes

...in/java/com/hedera/services/bdd/suites/regression/system/QuiesceThenMixedOpsRestartTest.java Outdated Show resolved Hide resolved

AlexKehayov dismissed gkozyryatskyy’s stale review via 336daf4 March 26, 2026 22:32

AlexKehayov marked this pull request as draft March 27, 2026 13:33

AlexKehayov mentioned this pull request Mar 27, 2026

chore: temporary disable quiesceAndThenRestartMixedOps #24578

Merged

add paired log assertion verb and pass env overrides through restart …

d2df7c6

…to fix quiescence Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

AlexKehayov force-pushed the 24398-quiescence-follow-up branch from 797302a to d2df7c6 Compare March 28, 2026 02:02

AlexKehayov and others added 12 commits March 28, 2026 04:29

refactor LogContainmentPairTimeframeOp

023032d

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

revert restartAtNextConfigVersion changes

1854bde

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

removed leaky and changed props in build gradle

e7eccca

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

run quiescence in a separate task

478de53

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

Merge branch 'main' into 24398-quiescence-follow-up

41dcd9f

added missing prop overrides to hapiTestQuiescence

2864f7b

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

Merge remote-tracking branch 'origin/24398-quiescence-follow-up' into…

df8c913

… 24398-quiescence-follow-up

enable quiescence in CI

6335853

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

move quiescence to time consuming

c31cb8a

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

move quiescence to a separate task

71c19ac

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

disable tss.forceHandoffs

1179dff

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

removed tss overrides

170aca3

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

AlexKehayov force-pushed the 24398-quiescence-follow-up branch from f78b067 to 170aca3 Compare March 30, 2026 01:06

AlexKehayov and others added 4 commits March 30, 2026 10:58

added rerun flags

56a492e

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

Merge branch 'main' into 24398-quiescence-follow-up

f9f8eca

made heartbeat tick after block closes and signs

6d7b67f

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

fix quiescence bug

28fc9c2

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

AlexKehayov added 3 commits March 30, 2026 18:36

resolved hanging

80348b2

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

revert ticks

874dade

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

skip epoch TCT in quiescence status check to prevent false DONT_QUIESCE

3a00302

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

AlexKehayov added 3 commits March 31, 2026 16:32

Merge branch 'main' into 24398-quiescence-follow-up

bc652dc

# Conflicts: # hedera-node/test-clients/build.gradle.kts

spotless

a342162

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

moved hapiTestQuiescence to Time consuming

9f64026

Signed-off-by: Alex Kehayov <aleks.kehayov@limechain.tech>

AlexKehayov force-pushed the 24398-quiescence-follow-up branch from 54942cf to 9f64026 Compare March 31, 2026 13:51

AlexKehayov and others added 5 commits April 1, 2026 10:43

Merge branch 'main' into 24398-quiescence-follow-up

6e05047

Merge branch 'main' into 24398-quiescence-follow-up

bd54e9a

isolate quiescence test into dedicated CI task without modifying core…

286da9c

… quiescence logic

Merge branch 'main' into 24398-quiescence-follow-up

c500d26

Merge branch 'main' into 24398-quiescence-follow-up

b2c5753

AlexKehayov changed the title ~~fix: increase assertion window in quiesceAndThenRestartMixedOps()~~ fix: resolve quiesceAndThenRestartMixedOps() flakiness Apr 2, 2026

Merge branch 'main' into 24398-quiescence-follow-up

583cd04

github-actions bot mentioned this pull request Apr 2, 2026

[Flaky Test] com.hedera.services.bdd.suites.integration.BlockNodeRewardsTests#activeNodeWithOneRegisteredBlockNodeGetsConsensusAndBlockReward #24711

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: resolve quiesceAndThenRestartMixedOps() flakiness#24567

fix: resolve quiesceAndThenRestartMixedOps() flakiness#24567
AlexKehayov wants to merge 29 commits intomainfrom
24398-quiescence-follow-up

AlexKehayov commented Mar 26, 2026 •

edited

Loading

Uh oh!

trunk-io bot commented Mar 26, 2026

Uh oh!

lfdt-bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

gkozyryatskyy left a comment

Uh oh!

Uh oh!

codecov bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

codacy-production bot commented Mar 31, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

AlexKehayov commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trunk-io bot commented Mar 26, 2026

Uh oh!

lfdt-bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

gkozyryatskyy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Node: HAPI Test (Misc) Results

Uh oh!

codacy-production bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Not up to standards ⛔

Uh oh!

github-actions bot commented Apr 2, 2026

⚠️ New Flaky Test(s) Detected — Action Required

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AlexKehayov commented Mar 26, 2026 •

edited

Loading

lfdt-bot commented Mar 26, 2026 •

edited

Loading

codecov bot commented Mar 26, 2026 •

edited

Loading

github-actions bot commented Mar 26, 2026 •

edited

Loading

codacy-production bot commented Mar 31, 2026 •

edited

Loading